Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to the Grading of Astrocytoma Tissues

摘要

We use partial class memberships in soft classification to model uncertainlabelling and mixtures of classes. Partial class memberships are not restrictedto predictions, but may also occur in reference labels (ground truth, goldstandard diagnosis) for training and validation data. Classifier performance is usually expressed as fractions of the confusionmatrix, such as sensitivity, specificity, negative and positive predictivevalues. We extend this concept to soft classification and discuss the bias andvariance properties of the extended performance measures. Ambiguity inreference labels translates to differences between best-case, expected andworst-case performance. We show a second set of measures comparing expected andideal performance which is closely related to regression performance, namelythe root mean squared error RMSE and the mean absolute error MAE. All calculations apply to classical crisp classification as well as to softclassification (partial class memberships and/or one-class classifiers). Theproposed performance measures allow to test classifiers with actual borderlinecases. In addition, hardening of e.g. posterior probabilities into class labelsis not necessary, avoiding the corresponding information loss and increase invariance. We implement the proposed performance measures in the R package"softclassval", which is available from CRAN and athttp://softclassval.r-forge.r-project.org. Our reasoning as well as the importance of partial memberships forchemometric classification is illustrated by a real-word application:astrocytoma brain tumor tissue grading (80 patients, 37000 spectra) for findingsurgical excision borders. As borderline cases are the actual target of theanalytical technique, samples which are diagnosed to be borderline cases mustbe included in the validation.